New York City Taxi and Limousine Commission data for Green Taxis This DS challenge is designed to evaluate your skills and intuition regarding a real world data problem. Data set: New York City Taxi and Limousine Commission trip records https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page
The yellow and green taxi trip records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts.
We'll use data from Green Taxis for September 2015.
Load data and analyze:
- Characterize the data and comment about its quality
- Explore and visualize the data e.g. a histogram of trip distance
- Find interesting trip statistics grouped by hour
- The taxi drivers want to know what kind of trip yields better tips. Can you build a model for them and explain the model?
- Pick one of the options below (Option 1) Find an anomaly in the data and explain your findings. (Option 2) Visualize the data to help understand trip patterns
Please submit the result in the form of runnable notebooks or scripts. A link to GitHub or other code repository would be great. Please let us know if we need to do anything special to run your notebook (install packages, get extra data etc.)
In [0]:
link="https://drive.google.com/open?id=1CFzqt-Ot1WPkW9pOGg_bYk4B7NJiXeGI"
_,id=link.split("=")
In [0]:
import warnings
warnings.filterwarnings('ignore')
Installing Libraries¶
In [131]:
!pip install geopy
!pip install wget
!pip install geopandas
In [0]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import io
import requests
import datetime as dt
import dask.dataframe as dask_dataframe
import dask.distributed
import scipy
import geopandas
import wget
from geopy.distance import vincenty
from shapely.geometry import Point
from sklearn import metrics
from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.model_selection import RandomizedSearchCV, train_test_split
In [0]:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
downloaded = drive.CreateFile({'id':id})
downloaded.GetContentFile('data_after_dask.csv')
data = pd.read_csv('data_after_dask.csv')
In [134]:
data.head()
Out[134]:
In [135]:
data.describe()
Out[135]:
In [136]:
distance_error = np.array(data["Trip_distance"])-np.array(data["distance_calculated"])
scipy.stats.describe(distance_error)
Out[136]:
- This cell below is imports LocationID, Borough, Zonename and service name as officially agreed by New York City Taxi and Limousine Commission data for Green Taxis
- The link below is the hyperlink of the data available on the NYC Taxi website [https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page]
In [137]:
url=r'https://s3.amazonaws.com/nyc-tlc/misc/taxi+_zone_lookup.csv'
taxi_zone = pd.read_csv(url)
pick_zone = taxi_zone.copy()
drop_zone = taxi_zone.copy()
print(pick_zone.columns)
pick_zone.columns = ['pickup_locationID','pickup_borough','pickup_zonename','pickup_servicezone']
drop_zone.columns = ['dropoff_locationID','dropoff_borough','dropoff_zonename','dropoff_servicezone']
pick_zone.head()
Out[137]:
In [138]:
drop_zone.head()
Out[138]:
In [139]:
result = pd.merge(data,pick_zone,left_on='pickup_zone', right_on='pickup_locationID',how='left')
result.head()
Out[139]:
In [140]:
result = pd.merge(result,drop_zone,left_on='dropoff_zone', right_on='dropoff_locationID',how='left')
result.head()
Out[140]:
In [0]:
result.drop(columns=['pickup_locationID','dropoff_locationID'],inplace=True)
In [142]:
result.head()
Out[142]:
In [143]:
result["pickup_servicezone"].value_counts()
Out[143]:
In [144]:
result["pickup_borough"].value_counts()
Out[144]:
In [0]:
def get_hours(result,column_name):
"""
Getting the hour of the day
Input:
result: DataFrame
Column_name: column from which hours have to be extracted
Output:
Pandas series of the hours
"""
return result[column_name].dt.hour
def get_day(result,column_name):
"""
Getting the weekday of the day
Input:
result: DataFrame
Column_name: column from which weekday have to be extracted
Output:
Pandas series of the weekday
"""
return result[column_name].dt.weekday_name
result['pickup_date'] = pd.to_datetime(result['pickup_date'], errors='coerce')
result['dropoff_date'] = pd.to_datetime(result['dropoff_date'], errors='coerce')
result['pickup_hr'] = get_hours(result,'pickup_date')
result['dropoff_hr'] = get_hours(result,'dropoff_date')
result['pickup_day'] = get_day(result,'pickup_date')
result['dropoff_day'] = get_day(result,'dropoff_date')
In [146]:
result.head()
Out[146]:
Characterize the data and comment about its quality¶
After getting to understand the description of the data. There are couple imperfection in data quality
- Considering the location of New York
- The latitude of any location (either pickup or drop) should be positive.
- The longitude of any location (either pickup or drop) should be negative.
- These rows can be removed by filtering rows having valid pickup/dropoff zones
- Passenger count can't less than 0
- Distance should in (0,Threshold).
- Threshold = 4 Std from the median vlaue
- All amounts($ - Fare_amount, Extra, MTA_tax, Tip_amount, Tolls_amount, improvement_surcharge, Total_amount) can't be negative
- Trip duration should in (0,Threshold).
- Lets sat threshold=200 minutes
We will have to clean the data based on which task we are accomplishing
In [0]:
data = result.copy()
Removing rows with location outside New York¶
In [148]:
condition = data["dropoff_zone"].isnull() | data["pickup_zone"].isnull()
print("Data having pickup zone or drop zone outside New York:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
Removing rows with number of passenger negative¶
In [149]:
condition = data["Passenger_count"].isnull() | data["Passenger_count"]<=0
print("Data with wrong number of passendger:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
Removing rows with negative amounts¶
In [150]:
condition = (data['Fare_amount']<2.5)|(data['Extra']<0)|(data['MTA_tax']<0)|(data['Tip_amount']<0)|(data['Tolls_amount']<0)|(data['improvement_surcharge']<0)|(data['Total_amount']<2.5)
print("Data with wrong amounts:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
Removing rows with very large trip duration or negative¶
In [151]:
data['trip_time'] = (data['dropoff_date'] - data['pickup_date']).astype('timedelta64[m]')
condition = (data['trip_time']<=0) | (data['trip_time']>200)
print("Data with wrong trip durations:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
Adding average speed of the ride column¶
- Column name: trip_speed
- miles/hour
In [0]:
#Speed
data['trip_speed'] = data['Trip_distance']*60 / data['trip_time']
Adding Tip percentage column¶
- Column name: Tip_percent_amount
- Tip(%)*100/Fare_amount($)
In [0]:
#Tip percentage of Fare amount
data['Tip_percent_amount']=data['Tip_amount']*100/data['Fare_amount']
Adding Tip per time column¶
- Column name: Tip_percent_time
- Tip($)*100/trip time(in mins)
In [0]:
#Tip percentage of trip duration
data['Tip_percent_time']=data['Tip_amount']*100/data['trip_time']
Removing rows with very large proporation of tips¶
In [155]:
condition = (data['Tip_percent_amount']>50)
print("Data with wrong amounts:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
Removing rows with impossible speeds¶
In [156]:
condition = (data['trip_speed']>150) | (data['trip_speed']<0)
print("Data with wrong amounts:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
Trip distance analyis¶
In [157]:
plt.hist(data['Trip_distance'], 50, facecolor='blue', alpha=0.8)
plt.xlabel('Trip Distance in miles')
plt.ylabel('Probability')
plt.title(r'Probability density of Trip distance')
plt.show()
This clearly shows that this is log-normal distribution. Moreover we will have to remove outliers before we go forward with the analysis.¶
- This is very positively skewed distribution
- to remover outlier condier data from between Min and Mean-4*Standard _deviation
In [158]:
from scipy.stats import lognorm
fig,ax = plt.subplots(1,3,figsize = (20,5))
dist = data['Trip_distance']
ax[0].hist(dist,bins=range(0,120,1),facecolor='green', alpha=0.8)
ax[1].hist(dist,bins=range(0,120,1),facecolor='green', alpha=0.8)
ax[2].hist(dist[~((dist-dist.median()).abs()>4*dist.std())],bins=range(0,16,1),facecolor='green', alpha=0.8)
ax[0].set_xlabel('Trip Distance')
ax[0].set_ylabel('Number of trips (Count - Scale)')
ax[0].set_title('Histogram - Trip Distance')
ax[1].set_xlabel('Trip Distance')
ax[1].set_ylabel('Number of trips (log - Scale)')
ax[1].set_yscale('log')
ax[1].set_title('Histogram - Trip Distance with log scale')
ax[2].set_xlabel('Trip Distance')
ax[2].set_ylabel('Number of trips')
ax[2].set_title('Histogram - Trip Distance without outliers')
# Fit line
scatter,loc,mean = lognorm.fit(data["Trip_distance"].values,scale=data.Trip_distance.mean(),loc=0)
pdf_fitted = lognorm.pdf(np.arange(0,12,.2),scatter,loc,mean)
ax[2].plot(np.arange(0,12,.2),1400000*pdf_fitted,'r')
ax[2].legend(['data','lognormal fit'])
plt.show()
The Trip Distance is positively asymmetrically skewed distributed. The skewness is due to high concentration of the rides towards lower bound(=0). This is a structure of a lognormal distribution.
- Left graph, density histograms. Entire dataset after intuitive data removal above
- Middle, density axis(y) is converted to logarithmic scale
- Right graph, outliers are removed. Outliers: points located further than 4 standard deviations from the median
The trips are not random becuase its not normal distribution.
Visualization and data analysis¶
In [0]:
def pickup_groupby(data):
"""
Groupby the data by column='pickup_zone'. Operator: Count
Input:
data: DataFrame, Taxi data after joined with taxi zone data
Output:
DataFrame with index='pickup_zone'
"""
count_pickup_zone = data.groupby('pickup_zone').count()
count_pickup_zone.columns=["count_pick_zone"]
return count_pickup_zone
def dropoff_groupby(data):
"""
Groupby the data by column='dropoff_zone'. Operator: Count
Input:
data: DataFrame, Taxi data after joined with taxi zone data
Output:
DataFrame with index='dropoff_zone'
"""
count_dropoff_zone = data.groupby('dropoff_zone').count()
count_dropoff_zone.columns=["count_dropoff_zone"]
return count_dropoff_zone
In [0]:
import bokeh, bokeh.plotting, bokeh.models
from bokeh.io import output_notebook, show, output_file, curdoc
from bokeh.layouts import row, column
from bokeh.models import ColumnDataSource, Slider
from bokeh.plotting import figure
output_notebook()
def bokeh_map(df):
"""
Show New york city with Heat map
Input:
data: DataFrame
Output:
show()
"""
gjds = bokeh.models.GeoJSONDataSource(geojson=df.to_json())
tool_list = ['pan','wheel_zoom','reset','hover','lasso_select', 'tap']
pickup = bokeh.plotting.figure(title="NYC Green Taxi Pickup", tools=tool_list,plot_width=500, plot_height=500)
dropoff = bokeh.plotting.figure(title="NYC Green taxi Dropoff", tools=tool_list,plot_width=500, plot_height=500)
blue = bokeh.models.LogColorMapper(palette=bokeh.palettes.Blues9[::-1],low=1, high=5000)
green = bokeh.models.LogColorMapper(palette=bokeh.palettes.Greens9[::-1],low=1, high=5000)
pickup.patches('xs', 'ys', fill_color={'field': 'count_pick_zone', 'transform': blue},
fill_alpha=1, line_color="gray",source=gjds)
dropoff.patches('xs', 'ys', fill_color={'field': 'count_dropoff_zone', 'transform': green},
fill_alpha=1, line_color="gray",source=gjds)
pickup.grid.grid_line_color = None
dropoff.grid.grid_line_color = None
hover_pickup = pickup.select_one(bokeh.models.HoverTool)
hover_pickup.point_policy = "follow_mouse"
hover_pickup.tooltips = u"""
<div>
<div class="bokeh_hover_tooltip">Name : @zone</div>
<div class="bokeh_hover_tooltip">Trips Start : @count_pick_zone</div>
</div>
"""
hover_dropoff = dropoff.select_one(bokeh.models.HoverTool)
hover_dropoff.point_policy = "follow_mouse"
hover_dropoff.tooltips = u"""
<div>
<div class="bokeh_hover_tooltip">Name : @zone</div>
<div class="bokeh_hover_tooltip">Trips Start : @count_dropoff_zone</div>
</div>
"""
blue_bar = bokeh.models.ColorBar(color_mapper=blue,
orientation='horizontal',
ticker=bokeh.models.FixedTicker(ticks=[5, 10, 50, 100, 500, 1000, 5000, 10000]),
formatter=bokeh.models.PrintfTickFormatter(format='%d'),
label_standoff=12,
border_line_color=None,
location=(0,0))
pickup.add_layout(blue_bar, 'above')
green_bar = bokeh.models.ColorBar(color_mapper=green,
orientation='horizontal',
ticker=bokeh.models.FixedTicker(ticks=[5, 10, 50, 100, 500, 1000, 5000, 10000]),
formatter=bokeh.models.PrintfTickFormatter(format='%d'),
label_standoff=12,
border_line_color=None,
location=(0,0))
dropoff.add_layout(green_bar, 'above')
p = row(pickup, dropoff)
show(p)
This cell below loads shape file (compatible with geopandas) to chart different taxi zones on Bokeh map
The link below is the hyperlink of the data available on the NYC Taxi website [https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page]
In [0]:
import ssl
import zipfile
ssl._create_default_https_context = ssl._create_unverified_context
wget.download('https://archive.nyu.edu/bitstream/2451/36743/3/nyu_2451_36743_WGS84.zip')
zipfile.ZipFile('nyu_2451_36743_WGS84.zip', 'r').extractall()
In [0]:
df_shape = geopandas.read_file('nyu_2451_36743_WGS84/nyu_2451_36743.shp').to_crs({'init': 'epsg:4326'})
df_shape = df_shape.drop(['Shape_Area', 'Shape_Leng', 'OBJECTID'], axis=1)
def generate_bokeh_data(df_shape,data):
bokeh_data = df_shape.merge(pickup_groupby(data[["pickup_zone",'dropoff_zone']]),
left_on='LocationID',
right_index=True,
how='left')
bokeh_data = bokeh_data.merge(dropoff_groupby(data[["pickup_zone",'dropoff_zone']]),
left_on='LocationID',
right_index=True,
how='left')
return bokeh_data
In [163]:
generate_bokeh_data(df_shape,data).head()
Out[163]:
In [164]:
bokeh_map(generate_bokeh_data(df_shape,data))
The Pickups are not evenly distributed. they are mostly concentrated on the regions touching the East River i.e. boundary between Manhattan and Queens or Brooklyn. There are not pickups in the Downtown Manhattan regions because of the regional aggrement between Green and Yello Taxi associations.
- Harlem has very high density of pickups as well as dropoffs
- Outskirts of NYC relatively use less of Green Taxis
Hourly study¶
In [0]:
# Mean / Median for hourly study
def hourly_distance(data):
hr_dist = pd.DataFrame(data[['Trip_distance','pickup_hr']].groupby(by='pickup_hr').mean())
hr_dist['Medium_trip_distance']=data[['Trip_distance','pickup_hr']].groupby(by='pickup_hr').median()
hr_dist.columns=['Mean_trip_distance','Median_trip_distance']
return hr_dist
In [166]:
plt.plot(hourly_distance(data));
plt.xlabel('Pickup hour')
plt.ylabel('Trip distance (miles)')
plt.title('Mean and median distance')
plt.legend(['Mean distance', 'Median distance'])
Out[166]:
The average trip distance is very high during early morning and late nights. This reflects the lifestyle of NYC.
- City that never sleeps
- People start commuting for the work early in the morning 3.Late Night event culture
In [167]:
data['pickup_borough'].unique()
Out[167]:
In [168]:
fig,ax = plt.subplots(2,3,figsize = (20,10),sharex=True)
fig.suptitle('Mean and median distance')
borough = data['pickup_borough'].unique()
for i in range(len(borough)):
ax[i//3,i%3].plot(hourly_distance(data[data['pickup_borough']==borough[i]]))
ax[i//3,i%3].set_title(borough[i])
ax[i//3,i%3].set_xlabel('Pickup hour')
ax[i//3,i%3].set_ylabel('Trip distance (miles)')
plt.show()
The main 4 boroughs show similar reuslt as of average NYC graph. Average ride distance for the pickps in Manhattan and Brooklyn is little higher than Bronx and Queens. Rides in Staten Island are realtively randomised, moreover we don't have large to comment something concrete on Staten Island and EWR
In [0]:
pick_count_borough = data[['pickup_hr','pickup_borough','pickup_zonename']].groupby(['pickup_borough','pickup_hr']).count()
In [170]:
pick_count_borough.loc['Bronx']
Out[170]:
In [171]:
labels = borough
width = 0.10 # the width of the bars
fig, ax = plt.subplots(1,1,figsize = (20,5))
for i in range(len(labels)):
x = pick_count_borough.loc[labels[i]].index
ax.bar(x + width*i*1.4,pick_count_borough.loc[labels[i]].pickup_zonename, width, label=labels[i])
# Add some text for labels, title and custom x-axis tick labels, etc.
ax.set_ylabel('Count of Taxi rides')
ax.set_xlabel('Pickup hour after midnight')
ax.set_title('Number of Taxi rides')
ax.set_xticks(np.arange(0, 24, step=1))
ax.legend()
plt.show()
This graph offer interesting insghts about Taxi usage in different boroughs.
- Brooklyn has most rides outside the normal woring hours
- Overall it trends number of rides peak between 8 PM and 12 AM
- If everything remaining same, if a taxi want to serve as mutch passengers as possible, Manhattan during daytime and Brooklyn during late evenings
Creating new variables¶
In [0]:
tip_amount_borough = data[['pickup_hr','pickup_borough','Tip_percent_amount']].groupby(['pickup_borough','pickup_hr']).mean()
In [173]:
tip_amount_borough.loc['Bronx']
Out[173]:
In [174]:
labels = borough
width = 0.10 # the width of the bars
fig, ax = plt.subplots(1,1,figsize = (20,5))
for i in range(len(labels)):
x = tip_amount_borough.loc[labels[i]].index
ax.bar(x + width*i*1.4,tip_amount_borough.loc[labels[i]].Tip_percent_amount, width, label=labels[i])
ax.set_ylabel('Tip/Fare amount - %')
ax.set_xlabel('Pickup hour after midnight')
ax.set_title('Tip percentage of Fare amount')
ax.set_xticks(np.arange(0, 24, step=1))
ax.set_ylim(top=20)
ax.legend()
plt.show()
General trend coming out is trips originating from the Brooklyn tends to pay higher tip percentage of the base fare amount. Next is Manhattan.
- This suggest aviod Bronx
- Try to get large number of pickups from Brooklyn region
In [0]:
tip_time_borough = data[['pickup_hr','pickup_borough','Tip_percent_time']].groupby(['pickup_borough','pickup_hr']).mean()
In [176]:
tip_time_borough.loc['Bronx']
Out[176]:
In [177]:
labels = borough
width = 0.10 # the width of the bars
fig, ax = plt.subplots(1,1,figsize = (20,5))
for i in range(len(labels)):
x = tip_time_borough.loc[labels[i]].index
ax.bar(x + width*i*1.4,tip_time_borough.loc[labels[i]].Tip_percent_time, width, label=labels[i])
ax.set_ylabel('Tip/(Time in minutes) - %')
ax.set_xlabel('Pickup hour after midnight')
ax.set_title('Tip percentage of Trip duration')
ax.set_xticks(np.arange(0, 24, step=1))
ax.set_ylim(top=20)
ax.legend()
plt.show()
In [0]:
tip_ride_borough = data[['pickup_hr','pickup_borough','Tip_amount']].groupby(['pickup_borough','pickup_hr']).mean()
In [179]:
tip_ride_borough.loc['Bronx']
Out[179]:
In [180]:
labels = borough
width = 0.10 # the width of the bars
fig, ax = plt.subplots(1,1,figsize = (20,5))
for i in range(len(labels)):
x = tip_ride_borough.loc[labels[i]].index
ax.bar(x + width*i*1.4,tip_ride_borough.loc[labels[i]].Tip_amount, width, label=labels[i])
ax.set_ylabel('Tip amount - $')
ax.set_xlabel('Pickup hour after midnight')
ax.set_title('Average Tip amount')
ax.set_xticks(np.arange(0, 24, step=1))
ax.set_ylim(top=4)
ax.legend()
plt.show()
Last 2 graphs also comfirms the bias that trips starting from Brooklyn have higher upside for the taxi drivers
In [181]:
data.columns
Out[181]:
In [182]:
fig,ax = plt.subplots(3,1,figsize = (20,15))
fig.suptitle('Variation of Tip with trip duration')
temp_df = data[['trip_time','Tip_percent_amount','Tip_percent_time','Tip_amount']]
temp_df['trip_time'] = pd.cut(temp_df['trip_time'], 50,labels=range(50))
temp_df = temp_df.groupby('trip_time').mean()
ax[0].bar(temp_df.index,temp_df['Tip_percent_amount'],color='orange')
ax[0].set_xlabel('Trip Duration - Max: 200 minutes')
ax[0].set_xticks(np.arange(0, 50, step=1))
ax[0].set_ylabel('Trip percentage of Fare amount')
ax[1].bar(temp_df.index,temp_df['Tip_percent_time'],color='blue')
ax[1].set_xlabel('Trip Duration - Max: 200 minutes')
ax[1].set_xticks(np.arange(0, 50, step=1))
ax[1].set_ylabel('Tip percentage of trip duration')
ax[2].bar(temp_df.index,temp_df['Tip_amount'],color='green')
ax[2].set_xlabel('Trip Duration - Max: 200 minutes')
ax[2].set_xticks(np.arange(0, 50, step=1))
ax[2].set_ylabel('Tip amount - $')
plt.show()
The above graphs depicts that trips with trip ride o between 30-50 mins will fetch the highest returns when calculated in term per distance or per time basis. This is very crucial analysis for the drivers
In [183]:
fig,ax = plt.subplots(1,3,figsize = (20,5))
fig.suptitle('Variation of Tip with numbr of passengers')
temp_df = data[['Passenger_count','Tip_percent_amount','Tip_percent_time','Tip_amount']]
temp_df = temp_df.groupby('Passenger_count').mean()
ax[0].bar(temp_df.index,temp_df['Tip_percent_amount'],color='orange')
ax[0].set_xlabel('Number of passengers')
ax[0].set_ylim(top=10)
ax[0].set_ylabel('Trip percentage of Fare amount')
ax[1].bar(temp_df.index,temp_df['Tip_percent_time'],color='blue')
ax[1].set_xlabel('Number of passengers')
ax[1].set_ylim(top=15)
ax[1].set_ylabel('Tip percentage of trip duration')
ax[2].bar(temp_df.index,temp_df['Tip_amount'],color='green')
ax[2].set_xlabel('Number of passengers')
ax[2].set_ylabel('Tip amount - $')
plt.show()
- Although rides with number of passengers less than 2 is in very large proportionate, number of passengers seems to be independent of Tip gathered.
- It sounds that Uber/Lyft pool service driver will earn higher tips if passengers follow similar behaviours as above.
In [0]:
def pickup_groupby_mean(data,column_name):
count_pickup_zone = data.groupby('pickup_zone').mean()
return count_pickup_zone[[column_name]]
def generate_bokeh_data_mean(df_shape,data,column_name):
bokeh_data = df_shape.merge(pickup_groupby_mean(data,column_name),
left_on='LocationID',
right_index=True,
how='left')
return bokeh_data
In [0]:
import bokeh, bokeh.plotting, bokeh.models
from bokeh.io import output_notebook, show, output_file, curdoc
from bokeh.layouts import row, column
from bokeh.models import ColumnDataSource, Slider
from bokeh.plotting import figure
output_notebook()
def bokeh_map_single1(df,column_name):
gjds = bokeh.models.GeoJSONDataSource(geojson=df.to_json())
tool_list = ['pan','wheel_zoom','reset','hover','lasso_select', 'tap']
pickup = bokeh.plotting.figure(title="NYC Green Taxi - Tip_percent_amount", tools=tool_list,plot_width=500, plot_height=500)
blue = bokeh.models.LogColorMapper(palette=bokeh.palettes.Blues9[::-1],low=1, high=20)
pickup.patches('xs', 'ys', fill_color={'field': 'Tip_percent_amount', 'transform': blue},
fill_alpha=1, line_color="gray",source=gjds)
pickup.grid.grid_line_color = None
hover_pickup = pickup.select_one(bokeh.models.HoverTool)
hover_pickup.point_policy = "follow_mouse"
hover_pickup.tooltips = u"""
<div>
<div class="bokeh_hover_tooltip">Name : @zone</div>
<div class="bokeh_hover_tooltip">Trips Start : @Tip_percent_amount</div>
</div>
"""
blue_bar = bokeh.models.ColorBar(color_mapper=blue,
orientation='horizontal',
ticker=bokeh.models.FixedTicker(ticks=[0.1,0.5,1,5,10]),
formatter=bokeh.models.PrintfTickFormatter(format='%d'),
label_standoff=12,
border_line_color=None,
location=(0,0))
pickup.add_layout(blue_bar, 'above')
p = row(pickup)
show(p)
In [186]:
bokeh_map_single1(generate_bokeh_data_mean(df_shape,data,'Tip_percent_amount'),'Tip_percent_amount')
Upper Manhattan and Brooklyn riders offer higher tip
In [0]:
import bokeh, bokeh.plotting, bokeh.models
from bokeh.io import output_notebook, show, output_file, curdoc
from bokeh.layouts import row, column
from bokeh.models import ColumnDataSource, Slider
from bokeh.plotting import figure
output_notebook()
def bokeh_map_single2(df,column_name):
gjds = bokeh.models.GeoJSONDataSource(geojson=df.to_json())
tool_list = ['pan','wheel_zoom','reset','hover','lasso_select', 'tap']
pickup = bokeh.plotting.figure(title="NYC Green Taxi - Tip_percent_time", tools=tool_list,plot_width=500, plot_height=500)
blue = bokeh.models.LogColorMapper(palette=bokeh.palettes.Greens9[::-1],low=1, high=20)
pickup.patches('xs', 'ys', fill_color={'field': 'Tip_percent_time', 'transform': blue},
fill_alpha=1, line_color="gray",source=gjds)
pickup.grid.grid_line_color = None
hover_pickup = pickup.select_one(bokeh.models.HoverTool)
hover_pickup.point_policy = "follow_mouse"
hover_pickup.tooltips = u"""
<div>
<div class="bokeh_hover_tooltip">Name : @zone</div>
<div class="bokeh_hover_tooltip">Trips Start : @Tip_percent_time</div>
</div>
"""
blue_bar = bokeh.models.ColorBar(color_mapper=blue,
orientation='horizontal',
ticker=bokeh.models.FixedTicker(ticks=[0.1,0.5,1,5]),
formatter=bokeh.models.PrintfTickFormatter(format='%d'),
label_standoff=12,
border_line_color=None,
location=(0,0))
pickup.add_layout(blue_bar, 'above')
p = row(pickup)
show(p)
In [188]:
bokeh_map_single2(generate_bokeh_data_mean(df_shape,data,'Tip_percent_time'),'Tip_percent_time')
Clearly Airports rides offer disproportionaly high tips when compared along with trip durations.
In [0]:
import bokeh, bokeh.plotting, bokeh.models
from bokeh.io import output_notebook, show, output_file, curdoc
from bokeh.layouts import row, column
from bokeh.models import ColumnDataSource, Slider
from bokeh.plotting import figure
output_notebook()
def bokeh_map_single3(df,column_name):
gjds = bokeh.models.GeoJSONDataSource(geojson=df.to_json())
tool_list = ['pan','wheel_zoom','reset','hover','lasso_select', 'tap']
pickup = bokeh.plotting.figure(title="NYC Green Taxi - Tip amount", tools=tool_list,plot_width=500, plot_height=500)
blue = bokeh.models.LogColorMapper(palette=bokeh.palettes.Oranges9[::-1],low=1, high=2)
pickup.patches('xs', 'ys', fill_color={'field': 'Tip_amount', 'transform': blue},
fill_alpha=1, line_color="gray",source=gjds)
pickup.grid.grid_line_color = None
hover_pickup = pickup.select_one(bokeh.models.HoverTool)
hover_pickup.point_policy = "follow_mouse"
hover_pickup.tooltips = u"""
<div>
<div class="bokeh_hover_tooltip">Name : @zone</div>
<div class="bokeh_hover_tooltip">Trips Start : @Tip_amount</div>
</div>
"""
blue_bar = bokeh.models.ColorBar(color_mapper=blue,
orientation='horizontal',
ticker=bokeh.models.FixedTicker(ticks=[0.1,0.5,1,2,5]),
formatter=bokeh.models.PrintfTickFormatter(format='%d'),
label_standoff=12,
border_line_color=None,
location=(0,0))
pickup.add_layout(blue_bar, 'above')
p = row(pickup)
show(p)
In [63]:
bokeh_map_single3(generate_bokeh_data_mean(df_shape,data,'Tip_amount'),'Tip_amount')
In [190]:
data.columns
Out[190]:
In [191]:
## rate code = 2 or 3
## pickup or dropoff location in JFK, LGA, or EWR
condition = (data['RateCodeID'].isin([2,3]))| (data['pickup_zonename'].isin(['JFK','LGA','EWR'])) | (data['dropoff_zonename'].isin(['JFK','LGA','EWR']))
#calculate airport and non airport trips number
data['airport_trip'] = 'Standard'
data['airport_trip'].loc[condition] = 'Airport'
print('Non - Airport Trips:', data['airport_trip'].value_counts()[0],'Airport Trips',data['airport_trip'].value_counts()[1])
In [192]:
airport_fare=data[['Total_amount','airport_trip']].groupby(by=['airport_trip']).mean()
airport_fare.columns=['mean_total_amount']
airport_fare['standard_deviation']=data[['Total_amount','airport_trip']].groupby(by=['airport_trip']).std()
airport_fare
Out[192]:
Tip prediction - Explainable way to taxi drivers¶
Basic level of cleaning is already done. Here I will examine each attribute and its logical sense
In [193]:
data.columns
Out[193]:
In [0]:
#No need of vendorID
data.drop(columns=['VendorID'],inplace=True)
In [195]:
#RateCodeID
data['RateCodeID'].value_counts()
Out[195]:
Most of the rides are with Standard Fares (~98%)
Next: Latitudes and longitudes are already analysed while performing visualization
In [196]:
data['Passenger_count'].value_counts()
Out[196]:
Field with zero passenger counts already removed in the previous analysis
In [197]:
data['Payment_type'].value_counts()
Out[197]:
In [198]:
data[data['Trip_distance']==0].shape
Out[198]:
Removing rows with Trip distance is less than 0.2 as well calculated is also less than 0.2 miles¶
In [199]:
condition = (data["Trip_distance"]<0.2) & (data["distance_calculated"]<0.2)
print("Data having small trip distance:",data[condition].shape)
data = data[~condition]
print("Remaining Data:",data.shape)
In [200]:
data['Tip_amount'].describe()
Out[200]:
Since the tip amount data is not available for the Cash payment. We will run the model on only Credit Card payment. However, this the model should also work on cash payments trips. This is becuase the distributions of other variables for payment=Credit and payment=cash are identical and of almost similar size
In [201]:
tip_condition = data['Payment_type']==1
tip_data = data[tip_condition]
tip_data.shape
Out[201]:
For Taxi Driver, Lets brain storm what are parameters He/She can judge for a trip beforehand to maximize the trip
- Passenger characteristics (here - number of passengers)
- Pickup location (here - Brough,Zone)
- Drop location (here - Borough,Zone). This is important because his/her next trip would probably start from dropping location
- Time of the day
- Day for the week
In [0]:
tip_data['time_range']=pd.Series(pd.cut(tip_data['trip_time'],
[0,15,30,60,200],
labels=['15 minutes','30 minutes','60 minutes','60+ minutes'],
include_lowest=True,
right=False,
retbins=False))
# Uber/Lyft ofer driver 3-hour intervals slot. That means most of the drivers prefer 3 hour slots
labels=['0-2','3-5','6-8','9-11','12-14','15-17','18-20','21-23']
tip_data['pickuphour_range']=pd.Series(pd.cut(tip_data['pickup_hr'], list(np.arange(0,25,3)), labels=labels,right=False))
In [0]:
data.drop(columns=['Store_and_fwd_flag','Ehail_fee'],inplace=True)
tip_data.drop(columns=['Store_and_fwd_flag','Ehail_fee'],inplace=True)
In [204]:
tip_data.columns
Out[204]:
In [205]:
fig, ax = plt.subplots(2,3,figsize=(20,10), sharey=True)
fig.subplots_adjust(hspace=.4)
cont_list =['pickup_borough','trip_time', 'pickup_hr', 'Passenger_count', 'Fare_amount', 'time_range']
for i, variable in enumerate(cont_list):
graph_data=tip_data.sample(500)
ax[i//3][(i+3) % 3].scatter(x=graph_data[variable], y=graph_data['Tip_percent_amount'],color='purple', alpha=.4)
ax[i//3][(i+3) % 3].set(title=variable, ylabel='Tip - % of the amount');
In [207]:
fig, ax = plt.subplots(2,3,figsize=(20,10), sharey=True)
fig.subplots_adjust(hspace=.4)
cont_list =['pickup_borough','trip_time', 'pickup_hr', 'Passenger_count', 'Fare_amount', 'time_range']
for i, variable in enumerate(cont_list):
graph_data=tip_data.sample(500)
ax[i//3][(i+3) % 3].scatter(x=graph_data[variable], y=graph_data['Tip_percent_time'],color='Orange', alpha=.4)
ax[i//3][(i+3) % 3].set(title=variable, ylabel='Tip - % of the amount');
The first question is what parameter is most important for the drivers From the different revolts from Uber/Lyft driver, many times they complain about not getting enough paid for the hours they put in.
Hence the intial hypothesis was there are 2 metric driver are concerned about
- Tip per unit time
- Tip % of the fare amount
The above 12 graphs suggest that, Tip are very likely related with fare amount rather than Tip per unit time. Clearly frund om graphs with Trip time, Pickup hour and fare amount vagues suggest that tipping culture is around 20% of the base fare
Hence we apply the model on Tip % amount to understand the tipping patterns
In [208]:
tip_data['pickup_servicezone'].value_counts()
Out[208]:
In [209]:
tip_data.head(5)
Out[209]:
In [0]:
# Import necessary libraries to build model
import statsmodels.api as sm
from statsmodels.tools.eval_measures import mse
from statsmodels.formula.api import ols
from sklearn.preprocessing import PolynomialFeatures,StandardScaler
import sklearn.model_selection
import sklearn.metrics
from sklearn_pandas import DataFrameMapper
In [0]:
tip_data_wo_dummy = tip_data.copy()
In [0]:
# create dummy variables
tip_data = pd.get_dummies(tip_data, columns=['Passenger_count','pickup_borough','pickup_servicezone',
'dropoff_borough','dropoff_servicezone','pickup_day','airport_trip',
'time_range','pickuphour_range'])
In [213]:
tip_data.columns
Out[213]:
In [0]:
mapper = DataFrameMapper([(['Fare_amount', 'Extra','Tolls_amount', 'MTA_tax','improvement_surcharge', 'trip_speed','Trip_distance'],[StandardScaler()]),
(['Passenger_count_1',
'Passenger_count_2',
'Passenger_count_3',
'Passenger_count_4',
'Passenger_count_5',
'Passenger_count_6',
'Passenger_count_7',
'Passenger_count_8',
'Passenger_count_9',
'pickup_borough_Bronx',
'pickup_borough_Brooklyn',
'pickup_borough_Manhattan',
'pickup_borough_Queens',
'pickup_borough_Staten Island',
'pickup_servicezone_Airports',
'pickup_servicezone_Boro Zone',
'pickup_servicezone_Yellow Zone',
'dropoff_borough_Bronx',
'dropoff_borough_Brooklyn',
'dropoff_borough_EWR',
'dropoff_borough_Manhattan',
'dropoff_borough_Queens',
'dropoff_borough_Staten Island',
'dropoff_servicezone_Airports',
'dropoff_servicezone_Boro Zone',
'dropoff_servicezone_EWR',
'dropoff_servicezone_Yellow Zone',
'pickup_day_Friday',
'pickup_day_Monday',
'pickup_day_Saturday',
'pickup_day_Sunday',
'pickup_day_Thursday',
'pickup_day_Tuesday',
'pickup_day_Wednesday',
'airport_trip_Airport',
'airport_trip_Standard',
'time_range_15 minutes',
'time_range_30 minutes',
'time_range_60 minutes',
'time_range_60+ minutes',
'pickuphour_range_0-2',
'pickuphour_range_3-5',
'pickuphour_range_6-8',
'pickuphour_range_9-11',
'pickuphour_range_12-14',
'pickuphour_range_15-17',
'pickuphour_range_18-20',
'pickuphour_range_21-23'], None)])
In [0]:
X_train, X_test, y_train, y_test = train_test_split(mapper.fit_transform(tip_data), tip_data['Tip_percent_amount'], test_size=0.2, random_state=42)
Linear Regression¶
In [216]:
model = sm.OLS(y_train, sm.add_constant(X_train)).fit()
predictions = model.predict(sm.add_constant(X_test))
model.summary()
Out[216]:
In [218]:
from sklearn.metrics import explained_variance_score
print('RMSE=',np.sqrt(mse(predictions, y_test)))
print('R-square=',explained_variance_score(y_test, predictions))
The linear regression model suggest the following insights
- Only few attributes affect the tip percentage
1. Fare_amount 2. Extra 3. Tolls_amount 4. MTA_tax 5. improvement_surcharge 6. trip_speed 7. Trip_distance - The p values of all the other attributes are way higher depcited they dont contribute to the tip percentage
- The Beta coefficients of the above 7 parametrs is also low
- This suggestion that model is having very high capacity, hence overfit over the data
- The results also bring the thoughts regadring the obersvation of constant Tip percentage(~20%) obeserved from the data analysis
Let us try robust algorithms like Random Forest, Ridge to rank the features¶
In [219]:
from sklearn.linear_model import Lasso,LassoCV
lasso_model = LassoCV(alphas=[0.01,0.1,0.5,1,5,10,50,100]) #Regularization penalty
lasso_model.fit(X_train, y_train)
lasso_train_score = lasso_model.score(X_train,y_train)
lasso_test_score = lasso_model.score(X_test, y_test)
print('Train coefficient R^2=',lasso_train_score)
print('Test coefficient R^2=',lasso_test_score)
In [220]:
lasso_model.coef_
Out[220]:
The result from the regulaized linear regression also suggest similar insights as of linear regression
In [128]:
import xgboost
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import MinMaxScaler
rf = RandomForestRegressor(n_estimators = 50, random_state = 42)
rf.fit(X_train, y_train);
predictions = rf.predict(X_test)
errors = abs(predictions - y_test)
print('Mean Absolute Error:', round(np.mean(errors), 2))
print('Explained Variance:',explained_variance_score(predictions,y_test))
In [0]:
X_train, X_test, y_train, y_test = train_test_split(tip_data_wo_dummy, tip_data['Tip_percent_amount'], test_size=1, random_state=42)
Baseline Model - Flat 20% Tip. The Purple graphs on the baove indicates the likely Tip percentage of the base fare amount¶
In [225]:
predictions = [20]*y_test.shape[0]
print('RMSE=',np.sqrt(mse(predictions, y_test)))
print('Mean Absolute Error:', round(np.mean(abs(predictions - y_test)), 2))
The heuristic 20% Tip precentage predict does better than all 3 models.
- This suggest people tip with similar percentages of that eating places (~18-20%)
Advice to Taxi driver to earn higher tips¶
- Choose airport rides over standard rides (Any JFK, LGA, Newark).
- Brooklyn should be prefred area of serive considering
- High tip percentage
- Tip per unit time
- Tip amount
- If you are not able to find the rides, you should operate in Upper Manhattan region during the day and Brooklyn(East River side) during late evenings
- Trips with 30-50 mins wold result in maximum tip earning considering you get enough passengers
- Early morning(between 5 AM and 8 AM) prefer taking longer trips, as number trips would be low and most of the customers would be travelling a long distance for work
- Late evenings(between 9 PM and 1 AM) prefer shorter rides, preferably in Brooklyn over Manhattan, to obtain large tip amount by doing frequenct short trips
- Prefer other regions over Staten Island and Bronx